InformatIon and CommunICatIon teChnology eduCatIon

This paper introduces the field of affective computing, and the benefits that can be realized by enhancing elearning applications with the ability to detect and respond to emotions experienced by the learner. Affective computing has potential benefits for all areas of computing where the computer replaces or mediates face to face communication. The particular relevance of affective computing to e-learning, due to the complex interplay between emotions and the learning process, is considered along with the need for new theories of learning that incorporate affect. Some of the potential means for inferring users’ affective state are also reviewed. These can be broadly categorized into methods that involve the user’s input, and methods that acquire the information independent of any user input. This latter category is of particular interest as these approaches have the potential for more natural and unobtrusive implementation, and it includes techniques such as analysis of vocal patterns, facial expressions or physiological state. The paper concludes with a review of prominent affective tutoring systems and promotes future directions for e-learning that capitalize on the strengths of affective computing. DOI: 10.4018/jicte.2012100107 76 International Journal of Information and Communication Technology Education, 8(4), 75-89, October-December 2012 Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. more tightly integrated with the day to day physical world, developments in this area are applicable to a vast array of situations such as embedded applications, information appliances, vehicles and so forth. There is evidence that emotion has an impact on the speed at which information is processed (Öhman, 2001) and whether it is attended to (Anderson, 2001; Vuilleumier, 2001). Emotion also has a relation to motivation in that evaluations or feelings regarding the current situation will largely determine the action that is taken in response. Therefore, emotions are often precursors of motivations (e.g., Oatley, 1992). Memory is also impacted by emotional state, and again there are many mechanisms by which this can occur. The Processing Efficiency theory (Eysenck & Calvo, 1992) suggests that emotions can utilize cognitive resources that would otherwise be used for processing new information; for example in the case of anxiety, intrusive thoughts may compete with the cognitive task and result in a decrease in performance. Thus, an area which can benefit greatly from affective computing is education. The fact that interaction with computers is a fundamental part of study in most disciplines, coupled with the cognitive and emotional journey that all learners experience makes e-learning an ideal candidate for affective computing developments. Intelligent tutoring systems attempt to emulate a human tutor by providing customized feedback or instruction to students. Whilst intelligent tutoring systems remain an active area of research, they have failed to achieve widespread uptake. A reason for this is the technical difficulty inherent in building cognitive models of learners and facilitating human-like communications (Reeves, 1998). The difference in learning performance between ideal one-toone tutoring conditions and other methods is known as the 2 Sigma problem (Bloom, 1984). Research on expert human tutors indicates that ‘expert human tutors devote at least as much time and attention to the achievement of affective and emotional goals in tutoring, as they do to the achievement of the sorts of cognitive and informational goals that dominant and characterize traditional computer based tutors’ (Lepper & Chabay, 1988, p. 242). Given the apparent link between cognition and affect, it may be argued that for an intelligent tutoring system to emulate a human tutor successfully there should be some consideration of affective processes during learning. The inability of current intelligent tutoring systems to cater for the role of emotion in learning may to some extent explain the 2 Sigma problem in the context of computer based learning. It is hoped that the incorporation of affective components into e-learning development may therefore lead directly to improved pedagogical outcomes. Providing this vital form of affective feedback into intelligent tutoring and other applications should greatly improve their success. Cognitive Basis for Learning The past few decades have seen the rise of the personal computer to fill many varied roles as organizer, communicator, entertainer and of course, educator. Research in the area of learning has predominantly taken a cognitive view in which the mental processes are considered as they are involved in learning. Cognitive theory is a learning theory of psychology that attempts to explain human behavior by understanding the thought processes. Cognitive theory is based on the assumption that human beings are logical and will make rational choices. The field of cognitive psychology provides explanations for many of the underlying mental processes that occur during learning. Prominent in this field is the three stage information processing model (Atkinson & Shiffrin, 1968) shown in Figure 1. This multi-store model of memory proposes that incoming information from the environment is briefly captured in sensory memory, and that information that is interesting is more likely to go on from sensory memory to short term memory. If a particular piece of information needs to be retained, the learner then makes a conscious decision to work with it and to continue to process it. Information that the learner has deemed important is International Journal of Information and Communication Technology Education, 8(4), 75-89, October-December 2012 77 Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. eventually encoded to the long term memory for storage and later retrieval. More recently constructivism has gained ground; constructivists believe that learners’ reality is built upon their existing experiences and perceptions. What someone knows is grounded in perception of the physical and social experiences which are comprehended by the mind (Jonassen, 1991). However, in spite of the way in which learning theories may have evolved over time, they have shared the perspective that the human mind is viewed as an information processing tool, not unlike basic computer architecture. Perkins highlighted the compatibility between traditional cognitive theories and constructivism, stating ‘...information processing models have spawned the computer model of the mind as an information processor. Constructivism has added that this information processor must be seen as not just shuffling data, but wielding it flexibly during learning -making hypotheses, testing tentative interpretations, and so on’ (Perkins, 1992, p. 51). Cognitive theories however do not explain the role that emotions play, in spite of the substantial evidence that emotions influence cognitive processes (Pekrun, 2008). Norman (1981) cited the topic of emotion as one of the major challenges to cognitive theory. Some authors consider the information-processing metaphor as the source of this challenge; for example, Ortony, Collins and Clore (1990, p. 5) stated ‘This approach to cognition has been as noticeable in its failure to make progress on problems of affect as it has been for its success in making progress on problems of cognition.’ People cannot be viewed purely as tasksolving, goal driven agents, they also have other emotive reasons for their choices and behavior that drive the decision making process (Mandler, 1975). Lisetti (1999) claims that a large number of cognitive tasks are influenced by affective state, including organization of memory, attention, perception and learning. The same conclusion was reached by Picard (1997, p. x) who states that ‘emotions play an essential role in rational decision making, perception, learning and a variety of other cognitive functions’. Cognitive-Affective Theory Another important area of research considers the underlying affective or emotional states and how these interact with cognitive processes. The way in which affective states interact with memory, decision making and social behavior creates a challenge for cognitive theory (Andrade & May, 2004). Emotions may disrupt, slow down, organize or initiate cognitive processes, and different emotions can influence these mechanisms in different ways (Pekrun, 2002). There has been a strong bias toward the cognitive and rational within the field of computer science, as a result of the prevailing view that the sciences are the domain of rules and logic with little room for anything else (Picard, 1997). In this view, emotion would be considered more of a distraction than a benefit. This bias has been reflected in the development of e-learning software, as it would generally be developed by programmers rather than learning theorists or educators. Consequently, many of the benefits of research into human affect and emotion are not yet fully realized in e-learning software. In the field of e-learning, a popular theory describing how learners process and learn from computer based multimedia is Mayer’s (2001) Cognitive Theory of Multimedia Learning. This theory draws from the multi-store model Figure 1. Three stage information processing model (Atkinson & Shiffrin, 1968) 78 International Journal of Information and Communication Technology Education, 8(4), 75-89, October-December 2012 Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. of memory described above, and others, to form a unified theory of the various aspects of cognitive processing of multimedia content and provides guidelines for instructional developers to improve learning outcomes. Central to the theory are the concepts that the human cognitive processes include limited working capacity, dual channels for various types of material (sound/ images) and that the information is actively processed and assimilated by the learner (Mayer, 2001). Moreno (2006) extended this model to include the role of affect in learning and named it the Cognitive-Affective Theory of Learning with Media (Figure 2). Where it differs from the original model is in the inclusion of affective and motivational factors. This addition acknowledges the role of affect as a mediator for rational cognitive processes such as learning. According to this theory, the level of interest that the learner has in the material will correlate to learning benefits by influencing students to invest more effort in the task. Furthermore, some instructional methods may be more supportive than others therefore producing improved learning outcomes by improving the student’s feelings about their ability to complete the task (Moreno, 2006). The author discusses the effect of emotions such as anxiety or confidence, but this theory could potentially also apply to a wider range of more subtle emotional expressions. Cognitive psychologists are not the only ones recognizing the link between emotion and mental processes; emotion theorists have long recognized that emotion itself may have a cognitive component. Schacter and Singer (1962) are known for their 2-factor theory in which they argue that there is a cognitive determinant to emotion. Before this work, emotion was believed to reflect biologically determined responses, and this perspective evolved to the view that emotion was a consequence of cognitive process and that various external factors determine the emotion that would be felt (Andrade & May, 2004). What this implies is that cognition and emotion are deeply intertwined, and that future developments in affective applications must acknowledge the two way interaction between these two basic areas of human functioning. The Role of Affect in Learning Stein and Levine (1991) have identified a link between a person’s goals and emotions, and proposed a goal-directed, problem solving model. As with other theories of emotion that indicate that people like to maximize positive affective states, their model assumes that people attempt to assimilate information into their existing knowledge – when this information is new it results in arousal of the autonomic nervous system – this, in conjunction with a cognitive appraisal results in an emotional reaction. Therefore this model predicts that learning always occurs during an emotional episode. Kort, Reilly, and Picard (2001) have developed a model that links emotions and stages of learning in a four quadrant spiral (Figure Figure 2. Cognitive-affective theory of learning with media (Moreno, 2006) International Journal of Information and Communication Technology Education, 8(4), 75-89, October-December 2012 79 Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. 3). The learning process is broken up by two axes, vertical and horizontal to signify learning and affect. The learning axis contains labels to indicate a range from constructive learning at one end, to un-learning at the other. The affect axis ranges from negative to positive. When a learner is working through a task with ease, they will be in quadrant I, experiencing constructive learning and positive affect. As the material becomes harder or if they struggle, they would move through quadrants II, III, and finally IV At this point they may be uncertain how to progress, but as they acquire new insights and ideas they will ultimately progress back to quadrant I so that the spiral may continue as they acquire more knowledge. Goleman (1995) reported that expert teachers are able to recognize emotional states of students, and respond appropriately to positively impact learning. Whilst the way in which this is accomplished is not well documented, and may indeed differ between teachers, the foundation is still the same: to recognize negative affect or states that are detrimental to learning and to guide the learner into a more positive and constructive state. Csíkszentmihályi (1990) described an ideal learning state, which he called the zone of flow. In this state, time and fatigue disappear as the learner is absorbed and immersed in the task they are undertaking. When in a state of flow, people are absorbed in the activity and feel in control of the task and environment (Hsu & Lu, 2004). These characteristics of flow, are identical to what players experience when immersed and fully engaged in games (Chen, 2007), indeed games which create a flow experience are likely to be adopted, whilst others are discarded (Sherry, 2004). Thus educational games may also benefit from this effect, as the engagement and enjoyment of the learner is a catalyst to mediate their future learning and interest (Fu, Su, & Yu, 2009). Intelligent tutoring systems attempt to emulate the personalized instruction that a human teacher may provide by building an internal model of the students’ knowledge, abilities and progress. An e-learning system with these characteristics can have many advantages; for example being always available and potentially being able to provide more individual attention than in a traditional class based lesson. Figure 3. Model relating phases of learning to emotions (Kort et al., 2001) 80 International Journal of Information and Communication Technology Education, 8(4), 75-89, October-December 2012 Copyright © 2012, IGI Global. Copying or distributing in print or electronic forms without written permission of IGI Global is prohibited. Intelligent tutoring systems incorporating an emotional or affective model are known as affective tutoring systems. An affective tutoring system is thus any tutoring system that can adapt to perceived emotion. This may be to respond to any negative emotions being experienced by the learner, or to interact in a manner that is more natural and engaging for the learner. These systems have also been shown to be effective and result in increased learning (as compared experimentally to a non-affect sensing implementation), however are still not as effective as a one-to-one human tutor. Further work is required. For theories linking learning and affective states to be implemented into the development of affective tutoring systems an important consideration is the means by which the affective state can be inferred by the computer. The next section discusses the options that are available, and this is followed by a review of affective tutoring systems. Inferring the Learner’s Affective State Given a suitable model to map affective states to desired behaviors or outcomes, the (technological) challenge is how to detect or infer the emotional state of the learner in the first place. There are several approaches to this, each with their own strengths and shortcomings. One of the key issues surrounding the inference of affective state is the relationship between the underlying emotion and the observable expression or behavior which accompanies it. Schachter (1962) argued that the differentiation of emotion is not physical, but cognitive, and the data does support the fact that various observable signals may be common to a multitude of differing emotional states. Some signals are better than others for differentiating affective states, and one point which is agreed upon is that no single signal is a sufficient indicator of emotional response (Picard, 1997). Affective states are internal and involve cognitive processes and are therefore not directly accessible to anyone other than the one experiencing them. Therefore it is only the observable manifestation of the affective state that may be used for the process of inference. This is where the subtle, non-verbal indicators of underlying affect become especially useful. A further question is whether emotions may be categorized into discrete states, or whether they are dimensional constructs, which vary along a continuum with several components. According to discrete emotion theories, certain emotions like happiness, fear, sadness or interest are considered to be discrete, unique states that are experienced as the result of distinct causes (e.g., Izard, 1977); many discrete emotion theories share the idea that a specific set of emotions is more basic or primary than the other emotions. These emotions are related to action tendencies and will thus have a physiological referent. In dimensional models of emotions, it is assumed that emotions can be represented in terms of a number of component dimensions (e.g., Russell, 1980). This viewpoint has the benefit of removing the need to categorize emotional experience within pre-defined boundaries, and may thus allow for a more fine-grained level of description.